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ABSTRACT 

A study determined the viability and effectiveness of 
the Computerized Competent Speaker Evaluation System. A total of 659 
comments made by undergraduate students about the paper version of 
the Competent Speaker Speech Evaluation Form were subjected to two Q 
sorts. The 635 comments that emerged were incorporated into the 
evaluation system during its development. Subjects were taken from 
four recitation sections (each with 15 to 20 students) of an 
introductory public speaking course. Speeches made by the students in 
two control groups were evaluated with the traditional handwritten 
method using the printed version of the speech evaluation form. 
Speeches of students in the two experimental groups were evaluated 
with the computerized version of the form. Graduate teaching 
assistants evaluated the speeches. Results were invalid since raters 
scored speeches considerably differently. Results of a posttest 
questionnaire of students indicated that Lhey perceived the system to 
be viable and efficacious. Analyses of a journal kept during the test 
period and of posttest interviews with the graduate teaching 
assistants using the system provided additional insights into the 
computerized evaluation system. Findings suggest that the system is 
viable, but the system's effectiveness is not yet established due to 
problems of low inter rater reliability. (Contains 23 references and 5 
tales of data. Evaluation forms, criteria for use of forms, sample 
screens from the computerized systems, a sample printout, and cost 
estimates for the computerized systems are attached. (RS) 
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Abstract 

This study concerns the development and a quasi- 
experimental test of the Computerized Competent Speaker 
Evaluation System. The rationale for developing such a 
system is provided. Then the paper explains how a 
database of comments was collected, Q-sorted, and 
analyzed. Also, the construction of the computer 
software for the system is described. A quasi - 
experimental test of the system, examining its 
viability and efficacy as compared to traditional 
handwritten feedback, is described. The scores of 
students in a test group on speeches are compared with 
the scores of students in a control group using t 
tests. These results are shown to "oe invalid since the 
raters in both groups scored speeches considerably 
differently. The results of a posttest questionnaire 
of students involved in this study indicate that they 
perceived the system to be viable and efficacious. 
Analyses of a journal kept during the test period and 
of posttest interviews with the graduate teaching 
assistants using the system provided additional 
insights into the computerized evaluation system. 
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Immediate Computerized Feedback in the Evaluation of 
Undergraduate Public Speaking: 
Development of the Computerized Competent Speaker 
Evaluation System and Testing of its 
Viability and Efficacy 
It is safe to assert that the majority of teachers 
of public speaking desires to provide effective 
feedback to their students. In his classic question, 
Holtzman (1960) states it best: "The instructor of 
speech has one primary question to answer: What can I 
say (or write or do) that will result in this student's 
improving his communication ability?" (p. 1) . Also, 
Book (1985), in an article analyzing effective feedback 
on speeches, asserts that experienced speech teachers 
have written a great deal about what comprises 
effective feedback . 

The type of feedback referred to in the study 
reported in this paper is consistent with Clement and 
Frandsen's (1976) definition of "action-reaction," 
(pp. 11-28) in other words, metacotranunicat ion : 
deliberate suggestions about a speech given subsequent 
to the speech. It is these deliberate comments 
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f ol lowi ng t.ho speech t.hat Dedmon (1967) aanort.n nhoul d 
be given directly after the students speak or right 
before the class period ends. Dedmon cites Byers 
(1963) , who suggests that students learn the best when 
they are given "immediate and valid knowledge" of their 
failure or success (p. 139) . Although Byres (1963) is 
arguing for immediate oral feedback, a similar 
argument can be made for the value of immediate written 
feedback. However, effective immediate written 
feedback is more difficult to provide than delayed 
written feedback. 

The desire to provide effective written feedback, 
that is also immediate, to public speaking students 
presents speech instructors with a dilemma. Often, 
when providing written feedback, speech instructors 
keep the evaluation sheets and give them to the 
students on some following day. Of course, the obvious 
problem with this option is that it diminishes the 
impact of feedback given to students (Leubitz & 
Hamilton, 1990) . Another option is that the evaluator 
will provide handwritten comments at the end of the 
students' speeches, thus often supplying insufficient 
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feedback and frustrating both themselves and their 
students. Behnke and King (1984) point out that terse 
comments may be perceived as worthless by both students 
and instructor. 

Therefore, the question this study seeks to answer 
is as follows: "How can instructors of public speaking 
provide effective immediate written feedback?" 

It is an undisputed fact that microcomputers 
process data in a matter of minutes and often seconds 
(Fletcher, 1992) . Buxton and Henry (1982) assert that 
the microcomputer is the most significant new tool man 
has designed in centuries. 1 They suggest that among 
its many desirable features is its capability to 
provide immediate feedback and that all of its features 
can be used to develop basic skills, listing 
communication as one of the basic skills. Because of 
these desirable features, educators have sought to 
incorporate the use of the computer into their 
respective academic disciplines in order to educate 
students more effectively. For instance, a program in 
a Chicago elementary school used computers to provide 
immediate feedback to students on their math problems . 
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That study demonstrated significant gains in the 
students 1 scores (Swarm, 1991) . 

So, since public speaking evaluators have 
struggled with wanting to provide adequate immediate 
feedback to their students and since microcomputers are 
very capable of providing immediate feedback, one would 
ask, "Can computers be used to provide effective 
immediate written feedback to public speaking 
students?" This study will examine the viability and 
efficacy of providing immediate computerized feedback 
to public speakers. Simply put, viability is 
operat ionalized as whether or not the coniput ei i zed 
system is perceived to work. Efficacy refers to 
whether or not the computerized system enables students 
to improve in public speaking skills and/or grades 
earned for speeches. 

Method 

This project examined two hypotheses. Hypothesis 
One stated the following: If based on an appropriate 
model of speech evaluation and a scientific devel- 
opmental method, a viable computerized speech 
evaluation system can be developed. This hypothesis 
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was explored by an examination of the following: (a) 
the documented experience of two raters using the pilot 
Computerized Competent Speaker Evaluation System as 
they provided feedback throughout a test period, (b) 
specific items on a questionnaire given to students in 
a control and test group at the end of a test period, 
and (c) specific response? in an interview at the end 
of a test period of the two raters who used the 
Computerized Competent Speaker Evaluation System. 

Hypothesis Two stated the following: That if used 
by evaluators who have undergone training on the 
Computerized Competent Speaker Evaluation System, the 
system may prove to be efficacious when subjected to a 
quasi -experimental te9t. This hypothesis was 
investigated by an examination of the following: (a) 
more significant improvement in the grades of students 
who received immediate computerized feedback in 
comparison to students who received traditional 
handwritten feedback and (b) the feedback of students 
in both groups on certain questions on the 
questionnaire given to them at the end of the test 
period . 
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Summarily, this project involved the formation of 
a database of speech evaluation comments for the 
Computerized Competent Speaker Evaluation System, the 
development of the computerized form, and the quasi - 
experimental testing of this form. 
Development of the Computerized Competent Speaker 
Evaluation System 

Decision on prototype . Much consideration was 
given to which public speaking form should be used as 
the prototype of the computerized evaluation system. 
It was decided that this form should be as 
comprehensive as possible and based on current 
literature on competent public speaking. It also 
needed to have been tested and found valid, reliable, 
and free from biases. The form chosen for the 
prototype was "The Competent Speaker" Speech Evaluation 
Form (Morreale, Moore, Taylor, Surges-Tatum, & Hulbert- 
Johnson, 1994) . 

"The Competent Speaker" Speech Evaluation Form 
(Morreale et al., 1994) (see Appendixes A and B for 
form and criteria) assesses public speaking behavior, 
focusing on communication skills . Eight skills, 
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referred to as individual competencies, were determined 
at> necessary in order for a student to exhibit 
competency in public speaking. An assessment of the 
students* preparation skills can be provided by 
examining the first four competencies. Also, an 
assessment of the students 1 delivery skil Is , both 
nonverbal and verbal behaviors, can be obtained by 
examining the last four competencies. Further, a 
holistic/aggregate assessment can be obtained by a 
total of the individual scores on these eight 
competencies . Degrees of assessment (unsatisfactory, 
satisfactory, or excellent) within each of the eight 
competencies are allowed for (Morreale et al, 1994). 

For a 2 -year period "The Competent Speaker" Speech 
Evaluation Form (Morreale et al . , 1994) was tested for 
reliability, validity, and cultural biases. The 
traditional testing indicated that the speech 
evaluation form is reliable and valid (Morreale et al., 
1994). The Rasch Analysis (Rasch, 1960) was used to 
analyze the speech evaluation form for cultural biases. 
It was found to be free from cultural biases (Tatum, 
1991). Thus, as a result of the testing, the 
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evaluation form was found to be reliable, valid, and 
free from cultural biases. 

Data collection . The paper version of "The 
Competent Speaker" Speech Evaluation Form was used in 
the sophomore level public speaking course (COMM 210) 
at the University of Colorado at Colorado Springs. For 
the fall of 1990 and the spring of 1991, as the 
graduate teaching assistants used "The Competent 
Speaker," their comments were stored in a database. In 
addition to these, approximately 200 comments were 
borrowed from a dissertation on public speaking 
(McMahan, 1988) . 

This resultant pool of comments was compared with 
the fragmented elements of the definitions of the eight 
competencies in "The Competent Speaker" Speech 
Evaluation Form (See appendixes A and B for form and 
criteria) . If a minimum of three comments was not 
found for each fragmented element of each of the 
definitions, additional comments were developed from 
the criteria. Many irrelevant comments were discarded 
for redundancy or ambiguity. The final pool of 
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comments submitted to a first Q sort numbered 659 com- 
ments . 

Data sorting . It was decided that through two Q 
sorts these 659 comments would be placed in the proper 
competencies and levels of these competencies. The Q- 
Sort technique was developed independently by 
Stephenson (1935) and Thomson (1935) . The Q-Sort 
procedure calls for the participants to select sets of 
objects or comments relative to the concept being 
studied and to categorize these objects or comments. 
Brooks suggests that the purpose of a Q sort is to 
obtain a conceptual illustration of the sorters' 
perspectives on the concept being evaluated (1970). 

According to Brooks (1970), the Q sets used in 
both sorts in this project were of the structured 
nature. They were established to represent the eight 
competencies of "The Competent S peaker" Speech 
Evaluation Form . The 659 comments were placed in 
random order for the start of the first Q sort. The 
evaluators were asked to sort these comment s into 
competencies one through eight, with number nine 
representing a discard category. The result was that 
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635 comments of the original 659 comments were 
determined to have content validity in this first 0 
sort . 

Then in a second Q sort, evaluators were asked to 
sort these 635 comments into levels of unsatisfactory, 
satisfactory, or excellent. Again, they were given a 
discard category. As a result of this Q sort 10 
comments were discarded. Thus, 625 comments of the 635 
comments were determined to have content validity in 
the second Q sort. 

Statistical analysis of data . In both Q sorts, 
the interrater reliability of these four raters was 
determined with the Krippendorff K formula 
(Krippendorf f , 1980) . Further analysis, involving the 
identification of the modal score for each comment and 
the most deviant rater out of the four, shed light on 
which comments had to be rejected. Any comments that 
drew less than .50 agreement of the four raters were 
removed from the database. The number of comments 
remaining at the end of these two Q sorts was 625. 

Construction of computer software program . While 
the database of comments was being constructed and 
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refined, available literature on the structural de- 
velopment of similar public speaking computerized 
feedback systems was evaluated. Based on interviews 
with Ralph Behnke (Zautke, 1991a) and Chris Sawyer 
(Zautke, 1991b), who had experimented with computerized 
speech feedback, the decision was made to use the 
software package, HyperCard™, on the Macintosh™ 
computer system (see Appendix E for sample costs) . 
Doris Carey (Witter, 1992), of the School of Education 
University of Colorado at Colorado Springs, an expert 
on the use of computers in the classroom, was consulted 
about the system 1 s conceptual layout in relation to 
Macintosh™ screens (see appendix C for sample 
screens) . The 625 comments that emerged from the two 0 
sorts were incorporated into the Computerized Competent 
Speaker Evaluation System during its development. 
Procedure Followed for Th ft Quasi-Experimental Test 
of Computerized C flffiBfitfi ltt Speaker Evaluation _Syafcfim 

pro-iect va ria bles . The independent variable in 
this study is the type of feedback that the student 
speakers receive, that is, computerized or handwritten 
feedback. Dependent, quantitative variables are the 
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scores that students earned on their speeches and their 
attitudes toward the type of feedback they received, as 
reported on the posttest questionnaire. Qualitative 
outcome variables are the analysis of the results of 
post-test interviews with raters of the experimental 
group and analysis of the journal of observations kept 
during the training period. 

Sample population . The sample was taken from the 
introductory public speaking class (COMM 210) at the 
University of Colorado at Colorado Springs during 
spring semester, 1993. Four recitation sections were 
used, each containing approximately 15-20 students. 
Thus, the total sample size was approximately 60-80 
students. Two recitation sections were designated as a 
test group, and two sections were designated as a 

control group. 

Training of eval uators- All four evaluators were 
trained in the use of "The Competent S peaker" Speech 
Evaluation Form . The two evaluators of the test group 
were trained additionally in the use of Computerized 
Competent Speaker Evaluation System. Successive weekly 
training was held as often as possible on "The 
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Competent Speaker" Speech Evaluation Form and on the 
Computerized Competent Speaker Evaluation System. 
After the four evaluators would critique a speech, they 
would compare their ratings on each of the eight 
competencies and their total scores , thus , endeavoring 
to normalize their grading tendencies. 

Testing . The 15-20 students in both groups 
presented the four major required speeches of COMM 210 
at assigned times during the semester of testing, 
spring, 1993. All students were videotaped as standard 
procedure in the COMM 210 course. The students in the 
control group were evaluated with the traditional 
handwritten method, using "The Competent Speaker" 
Speech Evaluation Form . On a day subsequent to t he i r 
presentations, these students would examine the 
handwritten feedback and view the videotapes of their 
speeches in an oral f ommunication laboratory. 

The students in the experimental group were 
evaluated with the Computerized Competent Speaker 
Evaluation System. The graduate teaching assistants 
printed the evaluation forms after evaluating the 
students. Immediately following their presentations, 
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the students took their videotapes to an oral 
communication lab where the students picked up the 
computer printouts of the evaluations of their 
speeches, read them, and viewed their speeches (see 
appendix D for a sample printout) . 

Instrumentation . After the test period elapsed 
and all of the speeches had been evaluated, a 
questionnaire focusing primarily on viability and 
efficacy was given to the students in both the test 
group and the experimental group. This questionnaire 
had been developed to examine four underlying 
constructs that seemed relevant to providing feedback 
with the use of the computer in a public speaking 
classroom: (a) the time factor, (b) the personableness 
of the computerized system, (c) rater capability, and 
(d) impact on students' skills and grades. It was 
reasoned that the students * perceptions in these areas 
would reveal a lot about how viable and efficacious the 
computerized system was to the students. This 
questionnaire was examined for face validity (Carmines 
& Zeller, 1979) . 
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A second evaluation that was made subsequent to 
the test period was separate interviews with the two 
graduate teaching assistants who used the Computerized 
Competent Speaker Evaluation System on the Macintosh tm 
computers . Just as the students 1 questionnaire had 
focused on viability and efficacy, so too this 
interview form was designed to focus on these aspects 
of competent speech evaluation. Since the interview 
form was patterned after the questionnaire and the 
questionnaire had been evaluated for face validity 
(Carmines & Zeller, 1979) , it was assumed that the 
interview form would also have face validity. 

Statistical analysis of data . The difference 
between the mean scores of students in the two sections 
comprising the experimental group on their last speech, 
as compared to the mean scores of the students in the 
two sect ions comprising the test group on their last 
speech in the control group was analyzed using t tests. 
It was thought that the improvement or lack of 
improvement in the score3 earned on speeches given by 
the experimental group, when respectively compared with 
the scores earned on speeches given by the control 
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group, would indicate the efficacy of the computerized 
system. 

Another series of t tests was run on the scores of 
the students 1 speeches, so that the grading tendencies 
of the evaluators could be investigated. The mean 
scores of each of the four raters were compared 
statistically so that the mean score of the students 1 
speech scores awarded by each rater was compared with 
the mean score of the students' speech scores awarded 
by each of the other raters. Also, the tendencies of 
the two raters in the test group collectively, as 
compared to the tendencies of the two raters in the 
control group collectively, were examined 
statistically. 

In addition to the t tests that were used to 
analyze the data collected from the students.' speeches, 
a MAN CO V A was used to analyze the data collected in the 
posttest questionnaire and to factor in a confounding 
variable. That confounding variable was the students' 
perceptions of their grades for the course as reflected 
in question 15 on the questionnaire. These tests were 
used to examine the difference in the amount of 
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perceived like or dislike for computerized or 
handwritten feedback (viability) and the students' 
perceptions as to which type of feedback most impacted 
ability and grades the between the control group and 
the t.e«t. _qt oup (efficacy). Multiple analyses of 
variance were run so that the mean scores could be 
adjusted to account for the confounding variance caused 
by the students' perceptions of their grades for the 
course . 

Thematic analysis of dat a. The data collected 
from observations made during the test period were 
recorded in a journal. These data were examined for 
subordinate themes, constructs that were significant to 
all four raters and the students involved in this 
experiment. Following this, the subordinate themes 
were synthesized into major themes or fundamental 
construct, s . 

A fWM'tind qualitative informal thematic analysis of 
the data gleaned from the interviews of the two raters 
who used the Computerized Competent Speaker Evaluation 
System was used to examine the raters' perceptions of 
the viability and efficacy of system. The data that 
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were collected from the interviews were examined for 
subordinate themes, constructs that were important to 
the raters who used the system. 

Results 

Quantifiable Data 

Interrater reliability in the first (? sort was 
.83. The total number of comments entering the first Q 
sort was 659, and the total number of comments 
remaining was 635. These 635 comments went into the 
second Q sort in which the interrater reliability was 
.60. 

The differences between the means of the scores 
earned on the final speeches of the test group as 
compared to the means of the scores earned on the final 
speeches of the control group were examined with t 
tests. The difference of the mean score between both 
groups of students is 7.6 points, or .068 of the 112 
possibl e points . The t value on the final speech wan 
statistically significant (p<.004). 
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Insert Table 1 about here. 



However, Table 2 suggests a confounding of this 
finding. It shows an analysis of the four evaluators 
in terms of their mean scores on each of the four 
operc*h« % H of t.hn LoHt. period. Ratotn 1 and 2 are the 
test group raters. Raters 3 and 4 are the control 
group raters. 



Insert Table 2 about here. 



On the final speech, Rater 1, in the test group, 
rated an average of 14.5 points above Rater 4, in the 
control qtowp. Those 14 point n oqunt o to . 12 r > of thn 
112 total points possible. Both of the raters in the 
test group individually rated more leniently than those 
in the control group. Also, the mean of the test group 
as compared to the mean of the control group, which was 
was statistically significant , shows that the raters in 
the test group collectively rated 7.6 points higher 
than the collective rating of the control group raters. 
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In order to examine the students 1 attitudes about 
the viability and efficacy of the Computerized 
Competent Speaker Evaluation System, a questionnaire 
was developed and administered. The investigation of 
the findings of this questionnaire are presented in 
Tables 3-5. 



Insert Table 3 about here. 



Item 1 shows a very significant statistical 
finding, and items 5, 8, 13, and 14 show statistically 
significant findings (£<.05 and u<.001). The students' 
responses to items 1 and 13 indicate that they felt 
that the time frame in which feedback was given to them 
impacted both their graden and abilities. Theii 
responses to items 5, 8, and 14 indicate that (a) the 
use of the computer in the classroom did not bother 
them, (b) they felt neutral about the database, and (c) 
they felt that computerized feedback provides more 
objective and fairer feedback than handwritten 
feedback . 
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In order to look for constructs , a cor re la t ional 
statistic was run on the questionnaire data. Four 
clusters of items on the questionnaire exhibited 
significant statistical correlational validity and 
correlational face validity. To factor out how 
students- 1 perceptions of their grades in the class 
affected their responses, a multivariate analysis was 
run on these four clusters. The item on the 
questionnaire that contained the confounding variable 
was item 15, which asked students what they perceived 
their grades would be in the public speaking course. 

Table 4 shows the "Observed Means" and the 
"Adjusted Means" for the four clusters of items that 
correlated in the data from the posttest questionnaire 
of students involved in the experiment.' 



Insert Table 4 about here. 



Only a few chose a grade other than "A" for the 
public spoaking course in response to item 15 on the 
questionnaire. The adjusted means of the clusters 
above indicate that the students' responses to the 
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items comprising these clusters were only minimally 
affected by their perceived grades. 

Table 5 shows that the only cluster of items from 
the questionnaire to manifest statistical significance 
is the cluster of items 1, 9, 12, and 13. Two of the 
items comprising this cluster probed the viability, and 
two items probed the efficacy of the computerized 
system. When these four items are combined, the 
resultant cluster is, "Rater/Feedback/Time Frame = 
Improvement" or "The time frame in which the rater 
provided feedback to me and the type of feedback 
provided to me caused my public speaking ability to 
improve and were pleasing to me." The F value for this 
cluster of items on the questionnaire indicates that 
the difference between the adjusted mean answer of the 
test group (4.63) and the adjusted mean of the control 
group (3.55) is significant (p<.024). 



Insert Table 5 about here, 
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Qualitative Data 

The journal of observations made during the test 
period seemed to synthesize into these five categories: 
(a) Not enough training and norming occurred, (b) the 
computerized system was not being used for a large 
portion of the speeches presented in the test group, 
(c) various complaints about the functioning of the 
system were voiced by the test group raters, (d) a 
primary concern was that the computerized system was 
taking too long to use, and (e) logistical and practical 
considerations were recorded in the journal. 

Interviews were conducted at the end of the test 
period with the graduate teaching assistants who rated 
speeches in the test group. The results of these 
interviews clustered into the following groups: 
general comments about the experiment, the viability of 
the computerized system, and efficacy of the system. 
General comments about the computerized evaluation 
system include comments about the variety and 
understandibility of the comments and the benefit of 
immediate feedback. A number of other issues were 
addressed by the test raters. Both raters of the test 
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group answered affirmatively that they would use the 
Computerized Competent Speaker Evaluation System again. 

Other questions in the interview examined the 
viability, or work-ability, of the computerized 
evaluation system. Although one test group rater did 
not like the impersonableness of the system, the other 
was not bothered by this. Both raters liked the 
dataoase but said that it contained too many comments. 
Even though one rater said that his grading was not 
affected by the use of the system, the other rater said 
that her grading was fairer and more objective. 

The efficacy, the impact on skills and grades, of 
the computerized evaluation system was also examined by 
the questionnaire. The two test group raters agreed 
that the immediate computerized feedback helped their 
students develop into better public speakers than if 
they had provided handwritten feedback. However, one 
restricted this by saying that the assistance was not 
significant. One rater said that his students did 
receive better grades because they received immediate 
computerised feedback. 
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Discussion 

This discussion is organized into three parts: 
First, the quantifiable data are discussed, including 
results of the two Q sorts, results of t tests run on 
students' grades on speeches given during the test 
period, results of t tests run to compare the scores 
awarded by the raters, and results of analyses of the 
student questionnaire. Second, the qualitative data 
are discussed, including observations made during the 
test period and responses given during posttest 
interviews with the two graduate teaching assistants 
rating t ho test group. Third, conclusions are drawn 
from these discussions about the viability and efficacy 
of the computerized evaluation system. 
Quantitative Data 

The results of the first Q sort indicate that the 
raters rated reliably between themselves. They rated 
with .83 reliability. Krippendorf (1980), citing 
Brouwer, Clark, Gerbner, and Krippendorff (1969), 
writes that a reliability finding over .80 should be 
reported. The results of the second Q sort indicate 
that the raters sorted the comments with a .60 
reliability. However, Krippendorf (1980) writes that 



Computerized Speech Feedback 

28 

even tentative conclusions cannot be drawn from results 
below .63. 

The large degree of interrater disagreement that 
Rater 4 had with the other three raters and the lack of 
interrater agreement in, especially, the second 0 sort 
call attention to the need for training in "The 
Competent Speaker" Speech Evaluation Form . This 
training would need to emphasize the criteria of "The 
Competent Speaker" Speech Evaluation Form and relative 
semantic distinctions. The result of such training 
would be a more accurate database. If the present 
study is replicated, both Q sorts should be conducted 
again after this improved training of the raters would 
be conducted. If both are not conducted again, at 
least the second level Q sort should be repeated. 

The comparison of means and t test scores revealed 
that those students in the test group out -performed the 
students in the control group (see Table 1) . The t 
value for the final speech is statistically significant 
(.004). In other words, these results indicate that 
test -group students' grades on speeches did improve 
and, thus, their public speaking skills also improved. 
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These results seem to support the hypothesis of this 
study that suggests that a computerized evaluation 
system may prove to be efficacious for students when 
used by evaluators who have undergone training on that 
system . 

These results, however, were confounded by the 
actual design of the experiment. Since the number of 
available speech recitation sections was limited, 
different graduate teaching assistants taught the test 
and the control sections. This confounding variable 
was examined with an analysis of the means scores on 
students' speeches as awarded by each rater on the 
final speech (see Table 2) . These data call into 
question the data presented in Table 1. Thus, the 
efficacy of the computerized system was not fully 
demonstrated in this project. Therefore, since the 
test group raters scored significantly higher than the 
control group raters on the final speech, the findings 
in Table 1, which suggest that the computerized system 
may have aided students in the test group to score 
higher, are shown to be invalid. 
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Although no quantitative measures were run and no 
records have been kept that would show multiple 
semester means for these raters prior to, the test 
period, Rater 1 consistently exhibited a tendency to 
award higher scores for the speeches of students in her 
sections as compa7:ed to how other graduate teaching 
assistants rated their students 1 speeches. On the 
other hand, Rater 4 exhibited tendencies in the 
opposite direction. Such rater error was evident in 
the discussions during the nor^ing sessions. Also, the 
raters in the control group may have been trying to 
compensate for the relatively slow printing of the 
computerized evaluation system or their lack of 
proficiency with. the system. 

On the t tests (see Table 3) that were run on 
individual items on the posttest questionnaire, five 
individual items on the questionnaire showed 
statistical significance (1, 5, 8, 13, and 14). The 
test group students' responses to items 1 and 13 
indicate that they felt that the computerized system 
was efficacious. On items 5, 8, and 14 they indicated 
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that they perceived the computerized feedback system as 
viable . 

Since the differences between the observed mean 
and the adjusted mean (see Table 4) f y qr both groups in 
this study were only minimally affected by the 
students' perceived grades for the course, one can 
assume that their responses on the questionnaire 
accurately reflected their perceptions. 

Of the four clusters of items from the 
questionnaire that formed after the correlational 
statistic was run, one cluster manifested a statistical 
significant difference (see Table 5). Since students 
in the test group perceived the rater, method of 
feedback, and time frame in which feedback was provided 
to them as all correlating to contribute to improvement 
in their public speaking skills, the viability and 
efficacy of the computerized evaluation system are both 
supported . 
Qualitative Data 

The data from the journal observations indicate 
that more training on the Computerized Competent 
Speaker Evaluation System and more norming among the 
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raters are needed than was accomplished in this 
project. A more thorough knowledge of the criteria 
contained in "The Competent Speaker" Speech Evaluation 
Form is needed. Additionally, journal observations 
documented that only the final speech could be examined 
for statistical significance. The number of times that 
the computerized system was used to rate speeches in 
the test group on the first three speeches was 
insufficient for test purposes. Another issue 
suggested that too much time was being consumed as 
raters concluded their comments and as the system 
printed the evaluations. it was noted that a faster 
printer was needed. 

The answers to the general questions in the 
posttest interviews support journal observations that 
the system needs to be quicker and the raters needed 
more training. it is important that the raters 
commented that they liked the ability to make 
particular comments to students. Also, the raters 
provided suggestions for various changes in the system, 
itself. The journal observations and interview 
suggestions about increasing the speed of the system 
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and about additional training need to be researched in 
another study. 

The answers to the questions devoted to the 
viability of the computerized system also suggest that 
additional training may be needed to eliminate the 
feeling of impersonableness created by the system. The 
raters' responses to an item dealing with the potential 
increased objectivity and fairness are split. In 
general, the responses of the graduate teaching 
assistants about the viability of the computerized 
evaluation system suggest that they are ambivalent 
about it. However, both raters did say that they would 
use the computerized system again. 

The test raters' responses to the questions in the 
interviews that probed' the efficacy of the computerized 
evaluation system indicate they both felt that their 
students 1 abilities improved because the students 
received immediate feedback. Furthermore, one said 
that his students earned better grades because of 
immediate feedback. In general, the responses of the 
graduate teaching assistants indicate they felt that 
the system is efficacious. 
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Conclusion 

The results of this study, which endeavored to 
answer the question about how instructors of public 
speaking can provide effective immediate written 
feedback to their students after their speeches, 
indicate the following : 

1 . The first hypothesis , which asks if a viable 
computerized system can be developed, seems to have 
been supported* The journal observations indicate 
that, although the system had difficulties, it is 
viable, The students' responses on the posttest 
questionnaire reveal that they perceived it as viable. 
Also, the responses given in the posttest interview 
show that the test group raters perceived the system as 
viable . 

2. The second hypothesis, which asks if the 
system will prove to be efficacious, was not fully 
supported. The students 1 grades on speeches do not 
indicate that the system aided in better grades and 
increased skill. However, the students' responses on 
the posttest questionnaire reveal that the test group 
students perceived that their grades and skills 
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improved because they were given immediate computerized 
feedback . 

The use of the Computerized Competent Speaker 
Evaluation System is, therefore, determined to be a 
promising instructional strategy for improving public 
speaking instruct ion . 
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Appendix A 

Copy of "The Competent Speaker" Speech Evaluation Form 



E NT SPEAKER 
SPEECH PERFORMANCE EVALUATION FORM 



SPEAKER'S NAME: 

^VALUATOR'S NAME:_ 



EIQHT PUBUC SPEAKINQ COMPETENCIES 

Competency One 

CHOOSES AND NARROWS A TOPIC APPROPRIATELY 

FOR THE AUDIENCE AND OCCASION 
Comments: 



SPE AKING PERFORMANCE RATINGS 
Scoring- 



Competency Two 
COMMUNICATES THE THESISISPECIFIC PURPOSE IN A 
MANNER APPROPRIATE FOR AUDIENCE AND OCCASION 

Comments: 



Competency Thru 
PROVIDES APPROPRIATE SUPPORTING MATERIAL 
BASED ON THE AUDIENCE AND OCCASION 

Comments 



Competency four 
USES AN ORGANIZATIONAL PATTERN APPROPRIATE 
TO TOPIC. AUDIENCE. OCCASION. I PURPOSE 

Comments 



Competency Fin 

USES LANGUAGE THAT IS APPROPRIATE TO THE 

AUDIENCE AND OCCASION 
Comments: 



Competency Six 
USES VOCAL VARIE TY IN RATE. PITCH. A 
INTENSITY. TO HEIGHTEN AND MAINTAIN INTEREST 

Comments: 



Competency Seven 
USES PRONUNCIATION. GRAMMA*. 6 ARTICULATION 
APPROPRIATE TO THE DESIGNATED AUDIENCE 

Commentt: 



Competency ftoftf 

USES PHYSICAL BEHAVIORS THAT SUPPORT THE 

VERBAL MESSAGE 
Comments. 



Genet Mi Comments. 

' 1990 Or. SnenAyn Morrest* end SCAJCAT Subcommittee 



Summeirve Score ot Competencies:. 
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APPENDIX B 

Criteria for Use of "The Competent Speaker" Speech 

Evaluation Form 

"The Competent Speaker" Speech Evaluation Form 

Eight Public Speaking Competencies and Criteria for 
Assessment 

Competency One 

CHOOSES AND NARROWS A TOPIC APPROPRIATE TO THE AUDIENCE 
AND OCCASION. 

EXCELLENT - The speaker presents a topic and a focus 
that are exceptionally appropriate for the purpose, 
time constraints, and audience . 

[That is, the speaker* s choice of topic is clearly 
consistent with the purpose, is totally amenable to the 
time limitations of the speech, and reflects unusually 
insightful audience analysis.] 

SATISFACTORY = The speaker presents .a topic and a focus 
that are appropriate for the purpose, time constraints, 
and audience . 

(That is, the speaker's choice of topic is 
generally consistent with the purpose, is a reasonable 
choice for the time limitations of the speech, and 
reflects appropriate analysis of a majority of the 
audience . ] 

UNSATISFACTORY = The speaker presents a topic and a 
focus that are not approp>riate for either the purpose, 
time constraints , or audience . 

[That is, the speaker's choice of topic is 
inconsistent with the purpose, the topic cannot be 
adequately treated in the time limitations of the 
speech, and there is little or no evidence of 
successful audience analysis.] 
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(Appendix B continued) 



Competency Two 

COMMUNICATES THE THESIS/SPECIFIC PURPOSE IN A MANNER 
APPROPRIATE FOR THE AUDIENCE AND OCCASION. 

EXCELLENT = The speaker communicates a thesis/specific 
purpose that is exceptionally clear and identifiable . 

[That is, there is no question that all of the 
audience members should understand clearly, within the 
opening few sentences of the speech, precisely what the 
specific purpose/thesis of the speech is.] 

SATISFACTORY = The speaker communicates a 
thesis/specif ic purpose that is adequately clear and 
identif iable . 

[That is, at least a ma jori ty of the audience 
should understand clearly, within the opening few 
BontiMiroH of t h^ npeech, precisely what rh^ specific 
purpose/i h^sis of the speech is.) 

UNSATISFACTORY = The speaker does not communicate a 
clear and identifiable thesis/specific purpose. 

[That is, a majority of the audience may have 
difficulty understanding, within the opening few 
sentences of the speech, precisely what the specific 
purpose/ thesis of the speech is.) 
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(Appendix B continued) 



Competency Three 

PROVIDES SUPPORTING MATERIAL APPROPRIATE TO THE 
AUDIENCE AND OCCASION. 

EXCELLENT = The speaker uses supporting material that 
is exceptional in quality and variety. 

[That is, supporting material is unarguably linked 
to the thesis of the speech, and further is of such 
quality that it decidedly enhances the credibility of 
the speaker and the clarity of the topic] 

SATISFACTORY « The speaker uses supporting material 
that is appropriate in quality and variety. 

[That is, supporting material is logically linked 
to the thesis of the speech, and is of such quality 
that it adds a measurable level of interest to the 
npeech . ] 

UNSATISFACTORY = The speaker uses supporting material 
that is inappropriate in quality and variety. 

[That is, supporting material is only vaguely 
related to the thesis of the speech, and variety is 
either too great or too little to do anything but 
detract from the effectiveness of the speech.] 
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.(Appendix B continued) 



Competency Four 

USES AN ORGANIZATIONAL PATTERN APPROPRIATE TO THE 
TOPIC, AUDIENCE, OCCASION, AND PURPOSE, 

EXCELLENT = The speaker uses an exceptional 
introduction and conclusion and provides an 
exceptionally clear and logical progression within and 
between ideas . 

[That is, the introduction clearly engages the 
audience in an appropriate and creative manner, the 
body of the speech reflects superior clarity in 
organization, and the conclusion clearly reflects the 
content of the speech and leaves the audience with an 
undeniable message or call to action.] 

SATISFACTORY = The speaker uses an appropriate 
introduction and conclusion and provides a reasonably 
clear and logical progression within and between ideas. 

(That is, the introduction clearly engages a 
majority of the audience in an appropriate manner, the 
body of the speech reflects adequate clarity in 
organization, and the conclusion reflects adequately 
the content of the speech and leaves a majority of the 
audience with a clear message or call to action.] 

UNSATISFACTORY = The speaker fails to use an 
introduction or conclusion and fails to provide a 
reasonably clear and logical progression within and 
between ideas* 

[That is, the introduction fails to engage even a 
majority of the audience in an appropriate manner, the 
body of the speech reflects lack of clarity in 
organization, and the conclusion fails to reflect 
adequately the content of the speech and fails to leave 
even a majority of the audience with a clear message or 
call to action.] 
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(Appendix B continued) 



Competency Five 

USES LANGUAGE APPROPRIATE TO THE AUDIENCE AND OCCASION, 

EXCELLENT = The speaker uses language that is 
exceptionally clear, vivid, and appropriate . 

[That is, the speaker chooses language that 
enhances the audience's comprehension and enthusiasm 
for the speech, while adding a measure of creativity 
that displays exceptional sensitivity by the speaker 
for the nuances and poetry of meaning.] 

SATISFACTORY = The speaker uses language that is 
reasonably clear, vivid, and appropriate . 

[That is, the speaker chooses language that is 
free of inappropriate jargon, is nonsexist , is non 
racist, etc.] 

UNSATISFACTORY « The speaker uses unclear or 
inappropriate language . 

[That is, the speaker chooses inappropriate jargon 
or language which is sexist, racist, etc.] 
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(Appendix B continued) 



Compe tencv Six 

USES VOCAL VARIETY IN RATE, PITCH, AND INTENSITY 
(VOLUME) TO HEIGHTEN AND MAINTAIN INTEREST APPROPRIATE 
TO THE AUDIENCE AND OCCASION, 

EXCELLENT = The speaker makes exceptional use of vocal 
variety in a conversational mode. 

[That, is, vocalics are exceptionally and 
appropriately well-paced, easily heard by all audience 
members, and varied in pitch to enhance the message.] 

SATISFACTORY = The speaker makes acceptable use of 
vocal variety in a conversational mode. 

[That is, the speaker shows only occasional 
weakness in pace , volume , pitch, etc . , thereby not 
detracting significantly from the overall quality or 
impact of the speech.] 

UNSATISFACTORY = The speaker fails to use vocal variety 
and failfi to speak in a con versa t ionai mode. 

(Th.it. is, the speaker shows frequent weakness in 
controlling and adapting pace, volume, pitch, etc., 
resulting in an overall detraction from the quality or 



impact of the speech . ] 
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(Appendix B continued) 



Competency Seven 

USES PRONUNCIATION, GRAMMAR, AND ARTICULATION 
APPROPRIATE TO THE AUDIENCE AND OCCASION. 

EXCELLENT = The speaker has exceptional articulation, 
pronunciation, and grammar. 

(That is, the speaker exhibits exceptional 
fluency, properly formed sounds which enhance the 
message, and no pronunciation or grammatical errors.] 

SATISFACTORY = The speaker has acceptable articulation, 
with few pronunciation or grammatical errors. 

(That is, most sounds are properly formed, there 
are only minor vocalized disf luencies , and a few (1A2) 
minor errors in pronunciation and grammar.] 

UNSATISFACTORY = The speaker fails to use acceptable 
articulation, pronunciation, and grammar. 

(That is, nonfluencies and disfluencies interfere 
with the message, and frequent errors in pronunciation 
and grammar make it difficult for the audience to 
understand the message.] 
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(Appendix B continued) 



Competency Eight 

USES PHYSICAL BEHAVIORS THAT SUPPORT THE VERBAL 
MESSAGE. 

EXCELLENT = The speaker demonstrates exceptional 
posture, gestures, bodily movement, facial expressions, 
eye contact and use of dress. 

[That is, kinesic (posture, gesture, facial 
expressions, eye contact) and proxemic (interpersonal 
distance and spatial arrangement) behaviors and dress 
consistently support the verbal message and thereby 
enhance the speaker's credibility throughout the 
audience . ] 

SATISFACTORY « The speaker demonstrates acceptable 
posture, gestures, facial expressions, eye contact and 
use of dress* 

[That is, kinesic (posture, gesture, facial 
expressions, eye contact) and proxemic (interpersonal 
distance and spatial arrangement) behaviors and dress 
generally support the message, with minor 
inconsistencies that neither significantly distract 
from the speaker's credibility with the audience nor 
interfere with the message.] 

UNSATISFACTORY = The speaker fails to use acceptable 
posture, gestures, facial expressions, eye contact and 
dress. 

[That is kinesic (posture, gesture, facial 
expressions, eye contact) and proxemic (interpersonal 
distance and spatial arrangement) behaviors and dress 
are incongruent with the verbal intent and detract from 
the speaker's credibility with the audience as well as 
distracting the audience from the speaker's message.) 
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Appendix C 
Copies of Sample Screens from the 
Computerized Competent Speaker Evaluation System 

'THE COMPETENT SPEAKER" 
COMPUTERIZED EVALUAT ION FORM 

Version Option: 

O Short Uersion (fewer comments) O Long Uerslon (many comments 

Scoring Option: 

O Normotiue Orluen Scoring O Criteria Orluen Scoring 



Directions: Move the mouse and place the index finger of the 
hand symbol inside the circle next to the name of 1) the length 
of tho form desired and click tne mouse once; 2) then do the 
same for the scoring option desired. 
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(Appendix C continued) 



THE COMPETENT SPEAKER SPEECH PERFORMANCE E VALUATION. 

ASSIGNMENT 

DATE Wod. 00^1994 

TIME 

PERFORMANCE RATING 

UNSAT SAT EXCaL 



SPEAKEfTSNAME 

SPEAKER'S SIO 

EVALUATORS NAME 



o i o r~o~ 



EIGHT PUBLIC SPEAKING COMPETENCIES 

COMPETENCY Q NE: Tooic Chosen and Nar rowed a pp. tot AndioncQ ft Occasion 
COMMENTS: 0 

COMPETEN CY TWO: The sis/Specifi c Piirrxvtt nor) lor Andift RW & Occasion. 
COMMENTS: 0 

COMPETENCY THREE: S t arting Material a no for AutfeoCT & QCCaSKXI 
COMMENTS. 0 

CQ_MP_EJ£NCY FOU R: Organiza tion app Topic. Audipnco Occ. & PutBQSfl 
COMMENTS: 0 

COMPETENCY HVE: [ aconXX^Yx ^jJ^TCQ ^OV^^ 

COMMENTS: 0 

COMPETE NCY SIX: Varmty In Rate Pitch, and Intensity non . for Aug. & 0C£. 
COMMENTS: 0 

COMPETENCY SEVEN: Pronunciation. Gram mar and Art, a do. for Aud. 
COMMENTS: 0 

COMPETE NCY EIGHT: Physical Behaviors that Sunnort the Verbal MfiSSaflfl 
COMMENTS: 0 

general COUVEMS: TOTAL SCORE OF COMPETEN CIES 0 



i j o i o 



° i 0 i 0 i 



( Start/Stop Time)(^aua (TrrhT)C^ULShMD^ 
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(Appendix C continued) 



COMPETENCY ONE: CHOOSES AND NARROWS A TOPIC APPROPRIATE TO THE AUDIENCE AND 
OCCASION. 


UNSATISFACTORY * The 
■oaikir present! ■ loplc md 

■ focui that iri not 

■ pproprlile for elthir the 
purpon, time constraint!, or 
•udlence. 


SATISFACTORY - The ipeiker 
preimti ■ topic end c focui 
that ara ipproprlite for tha 
purpoie, tlma comtrilnti, 
and audience. 


EXCELLENT « Tha ipiikir 
prannti a topic ind a focui 
that ara exceptionally 
■pproprlita for tha purpon, 
tlma comtrilnti, ind 
audience. 


Topic 

Fasinating \opK. but not iure 
how it relates to topic 
requirements 

Valuable topic . but this topic 
needed to be oM the topic list 

Focui 

Too much material 
Topic needs more focus 
I'm glad you get involved in 


o 

i 

1 

Ik 

n 

1 
1 

I 
o 


Topic 

Tha fa an Interesting topic. 
Thank you for sharing it. 

Topic is mundana 

Thought provoking topic 

Significant topic 

Good top*c for persuasion 

A challenging topic 

Interesting topic 

Focui 

Good narrowing of top»c 

Thoughtful topic need a tighter 
focus 

^ ' I Uj ' -* 


o 

m 

il 

A* 

i 

i 

l 

i. 

o 


Topic 

You have a good understanding 
of Issues that relate to your 
fellow man. 

Creative approach to topic 

You bring up worthwhile topics 
and discuss them intelligenUy. 

Fascinating topic 

Timely topic selection 

Great topic selection 

That was a difficult topic, and 
you covered all the major points 
well. 

Excellent onginat development 
of your topic 

Difficult but important 
topic-handled discreetly 

Excellent tope selection 

Focua 

Topic well selected and 
narrowed 
Novel approach to a familiar 


<> 

s 

ft , . 

it 

1 
1 

is 
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COMPETENCY TWO: COMMUNICATES THE THESISiSPECIFlC PURPOSE IN A MANNER 
APPROPRIATE FOR THE AUDIENCE ANO OCCASION. 



UNSATISFACTORY m The 
speaker doea nol 
comnunlcitt a clear end 
Identifiable thetlt/apecltic 
pu rpote. 



SATISFACTORY « The tpetker 
communlcatti * 
that It 'specific purpose that 
It adequately clatr and 
Identifiable. 



EXCELLENT » The speaker 
communicatee e 
Iheala/apeclflc purpose thai 
la exceptionally clear and 
Identifiable. 



Tha.lt 

Good introduction- not sure 
which focus pom! was thesis 

I don't understand what point 
you're trying to gat across 

Although the thesis «s 
included on your outline, it 
wasn't specifically included m 
presentation. 

I wasn't clear on your thesis 

Need to state thesis early and 
with emphasis 

Specific Purpoae 

Establish purpose earlier in 
piesentation 

Your exact purpose is 
unclear 

The porcer/ed purpose of 
speech not consistent with 
lopic 

Purpose was ccnfjr.ing 
Purpose was different than 
assignment 



Theafc 

Good thesis 

Emphasize thesis statement 



Specific Purpoae 

Interesting approach to topic 
Interesting purpose 
Speech purpose was 
accomplished. 

Purpose of your speech is 
consistent with topic 



o 

i 



Theele 

Clear ttatement of thesis 
Concise thesis statement 
Speech purpose was cloar and 

specific. 

Very clear thesis 

Your thesis was presented 

exceptionally clearly 

Very clear thesis statement 



o 



2*1 



Specific Purpose 

The purpose of your speech is 
very appropriate for audience 

Your purpose was 
extraordinarily appropriate for 
audience 

Your purpose was conveyed 
extremely well 



o 
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(Appendix C continued) 



COMPETENCY THREE: 
AND OCCASION. 



PROVIDES SUPPORTING MATERIAL APPROPRIATE TO THE AUDIENCE 



UNSATISFACTORY * The 
speaker uses supporting 
materiel that Is Inappropriate 
quality and variety, 



SATISFACTORY » The epeeker 
uses supporting meterlel thet 
le epproprlete tn qusllty end 
verlety 



EXCELLENT « The epeeker 
uses supporting meterlel that 
le exceptlonel In quellty end 
variety 



Supporting Meterlel 

Most information in the 
presentation is common 
knowledge. 

Need to define complex 
terms. 

Avoid passing pictures and 
objects around your audience 
while speaking. 

Use more authoritative and 
current evidence. 

Need to go beyond personal 
opmion-find expert support 
Remember the 30/70 rule 
Specific examples would have 
helped greatly to lighten focus 

Supporting matorial a bit 
disjointed 

Speoch aids would have 
helpod 

Too many examptes-choose 
*bost ol the best" 
Sourcos need to bo cited 
Cite more oxamplos to support 
the assertions. 



O Supporting Meterlel 

Concrete examples of personal 
examples 

Good selection of supporting 
material 
Make visual aids larger. 
Facts and figures could have 
been more meaningfully 
displayed In a chad. 

Good information; you really 
know your material. 

Good comparison of diftorent 
graphs 

Good use of narration 
Cite the source of your 
statistics. 

Effective handouts 
Good explanations of complex 
material 

Informative operational 
definition 

Need more specific examples 
Goc^ . ~:!©ty of evidence used 
lo support the thosis statement 
Good use of credible sources 



I 



f 

I 

Ik 



15 



Supporting Meterlel 
Clever use of cartoones 

Thought provoking 
information. 

Visual aid was well 
constructed. 

Slides were vory helpful 

Speech aid simple and 
effective-documentation dono 
wall 

You have obviously 
researched your topic well. 
Fascinating information 
Excellont exarnplos and 
research 

Purpose exceptionally 
supported and extended your 
topic 

Strong supporting material 
Dramatic uso ol data 
Excellent support and variety 
of support 

Outstanding visual aids 
Fluent supporting 
ma rA 
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(Appendix C continued) 



COMPTENCY FOUR: USES AN ORGANIZATIONAL PATTERN APPROPRIATE TO THE TOPIC, 
AUDIENCE. OCCASION, AND PURPOSE. 



UNSATISFACTORY * The 
speeker falls to us* sn 
Introduction or conclusion 
• nd («lls to provide s 
ressonslby clear end logical 
progression within and 
between Ideas. 



SATISFACTORY m Tha speaker 
uses en appropriate 
Introduction end conclusion 
and provides a resonalby 
cleer end logical progression 
within end between Ideee. 



EXCELLENT * The speaker 
usee en exceptions! 
introduction end conclusion 
end provides en exceptionally 
cleer end loglcel progression 
within end between Ideas. 



Orgsnlzetlonel Pettern 

Need work on strong 
introduction and transitions 

Tie conclusion specifically to 
mam points 

Lacked organization 

Development not totally 
logical 

You need to put more elfon 
into planning your speeches 
because 1 cant see n- jch 
improvement. 

Organizational «ogic was not 
cloar see me » <d we will work 
through this 

Transitions little bumpy 
disjointed 

Transitions would have helped 
to establish How between main 
points 

You need to study the correct 
form to use in outlining 



Orgenlzetlonel Psttern 

interesting introduction 

Don't say "Thank you" at the 
end of the presentation. 

Conclusion c tittle abrupt 

Preview needs to be more 
direct. 

Paraphrase and restate the 
complex concepts in the 
presentation. 

Your Ideas make sense and 
indicate careful preparation. 
Nice energetic introduction 

Good use of internal 
summaries 

A stronger preview would 
have reinforced the concepts of 
past and present 

The satisfaction step could use 
more development 
Clear call to action 

Make the call for action more 



i 
I 

1 



Br 

21 



Organizational Pettern 

Effoctive use of motivated 
sequence 

Effective introduction and 
organization and conclusion 

Super job at providing 
inlormationl 

Smooth transitions between 
ideas 

I understand. Your points aro 

well proven. 

Strong arguments 
Your points came across 

welt. 

Effective call to action 
Your organizational skills aro 
a great strength for you 
Creative introduction 
Exellent attention getting 
device and organization 
Excellent introduction 
Astute selection of points of 
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(Appendix C continued) 



COMPETENCY FIVE: USES LANGUAGE APPROPRIATE TO THE AUDIENCE AND OCCASION 



UNSATISFACTORY « The 
speaker uses unclear or 
Inappropriate language. 



SATISFACTORY > The speaker 
usee language that I* 
reasonably clear, vivid, and 
appropriate. 



EXCELLENT ■ The speaker 
uaaa languaga that la 
exceptionally claar, vivid, 
end approprlata. 



Languaga 

The language you chose 
clouded your message 

Your language was unclear in 
moaning and was not vivid 



Audience 

"You guys" is saxist and 
exclusionary 



o 



Language 

You explained the material 
well. 

You are very knowledgeable of 
relevant terms end demonstrate 
a wide vocabulary. 

Good word choices 

You used some words too 
many limes 

Good use ol comparison 

Your languaga was reasonably 
vivid in conveying meaning 

Your language lacked any 
poetical expression 

Your language was reasonably 
clear in conveying meaning. 

The language you chose tacked 
creativity 



Audience 

Your ideas coi'd be easily 
understood. 



'Oj Language 

Relevant use of analogy 
The language you chose 
showed a measure of 
creativity. 

Your language {word choice) 
made a stab at poetic 
expression. 

Effective use ot repetition 
You drew excellent pictures 
with words- great metaphors 
The language you chose added 
an excellent measure of 
creativity 

The words you chose to 
describe things painted vivid 
pictures for me 

Your language was 
exceptionally clear and sensitive 
to nuances of meaning. 

The language you chose was 
exceptionally poe'.cal. 

Audience 

Your terminology was 
well-lhoughi out and very 



I: 
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COMPETENCY SIX: USES VOCAL VARIETY IN RATE, PITCH, AND INTENSITY (VOLUME) TO 
HEIGHTEN AND MAINTAIN INTEREST APPROPRIATE TO THE AUDIENCE AND OCCASION. 



UNSATISFACTORY « Tha 
speaker fail* to uaa vocal 
variety and fall* to speak 
a convaraatlonal mo da. 



Vocal Varlaty 

You neod to work on youf 
vocalics as a whola 

Your vo»ca does not sound 
converstional 

The lack of a good pace, 
volume, and pitch detracted 
from your speech. 



Rate 

Try to slow down your rate of 
delivery. 

Slower rate of utterance 
needed 

Your fast spoaktng makes your 
listeners tense and nervous. 

Long pauses at tho wrong 
places ara ineffective 



SATISFACTORY . Tha apaakar 
makes acceptable use of vocal 
variety- In a conversational 
mode. - 



Vocal Variety 

Your use of good pace, volume 
and pitch helped your speech. 

You have a good 
conversational delivery and are 
easy to listen to. 

Good employment of vocal 
variety 

Try to use varwty in your 
voice 

You have a pleasant voice 
Rete 

Good pauses between major 
ideas 

Good use of silent pauses. 
Pause between major sections 
of the speech. 

Good speaking rate. 



o 



o 



EXCELLENT « Tha apeakar 
makes exceptional use of 
vocal varlaty In a 
convaraatlonal mode. 



Vocal Varlaty 

The excellent conversational 
quality of your voice was very 
pleasing. 

Your extraordinary use of 
pace, volume, and pitch greatly 
impacted your speech. 

Your voice is very dynamic 

Your use of vocal variety had 
a superb impact on your 
speech. 

Rale 

Your vocalics were 
appropriately well paced 

Your rate of speaking greatly 
enhanced the delivery of your 
speech. 

Tho pace of your vocalics 
strongly bolstered your 
speech 

The noticeable pause alter 
your main points was 
effective. 

Your vocalics are 
fiYranfinnallv anri nrwn 
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(Appendix C continued) 



COMPETENCY SEVEN: USES PRONUNCIATION, GRAMMAR, AND ARTICULATION APPROPRIATE 
TO THE AUDIENCE AND OCCASION. 




UNSATISFACTORY = Tha 
speaker falls to u»« 
■cc»pt»bl» irtlculitlon, 
pronunciation, and gnmmir. 


SATISFACTORY * Tha apaikar 
haa acceptable articulation, 
with lew pronunciation or 
grammatical errora. 


EXCELLENT = Tha tpaikar hi 
excaptlonaly articulation, 
pronunciation, end gnmmii 




Pronunciation 

Won\ on pronouncing each 
word so that words don't run 
together. 

Your regional pronunciations 
ot certain words distracted 
from your message. 

Your pronunciation of many 
words sounded incorrect. 

Grammar 

Pronoun/antecedo. 
disagreement is a flagrant 
error. 

Your improper grammar 
makes your speech confusing. 

Subject/vorb disagreement is 
a blatant error 

Your grammatical errors made 
understanding your spooch 

— »» 


<> 

m 

c 

f 

V 

% \ 

1 

<3 


Pronu nclatlon 

Your enunciation of many 
words needs to be sharpened. 

Pronounce your words 
clearly. 

Good enunciation 

Watch ending sounds o! 
words. 

Be secure in pronunciation o! 
names before using them. 

The pronunciation of most of 
your words was dear. 

Your pronunciation of one or 
two words sounded incorrect. 

Grammar 

Do not and sentences with 
prepositions. 

Your grammatical errors made 
understanding your speech 
slightly difficult. 

Awkward sentence structure 

Your grammar needed a little 
bit of cleaning up. 

You use good grammar when 


o 

1 

$ 

"is 

y 

AK 

A 

fit 

iV. 

<3 


Pronunciation 

Your pronunciation for att ol 
the words you used sounded 
correct. 

Your enunciation of words 
was very clear. 

Your pronunciation ot words 
enhanced your message. 

Grammar 

Your use of grammar was 
extraordinary. 

1 noticed no grammar errors 
m your speech 

Grammatically speaking, your 
speech was extremely clear 

r 


<> 
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(Appendix C continued) 



COMPETENCY EIGHT: USES PHYSICAL BEHAVIORS THAT SUPPORT THE VERBAL MESSAGE, 



UNSATISFACTORY * The 
speaker feile to ut« 
acceptable poatura, 
gestures, facial 
expressions, aya contact and 
raaa. 



SATISFACTORY « Tha apaakar 
demonstrates accaptabta 
poatura, gestures, facial 
expressions, aya contact and 
uaa ol draaa. 



EXCELLENT = Tha apaakar 
damonttratat exceptlonel 
poatura, gestures, bodily 
movement, facial 
expressions, aya contact 
uaa of draaa, 



and 



Phyalcal Behavior 

Don't distribute handouts until 
one is ready to use them. 

Your physical behaviors do 
not support your verbal 
message 

The non-verbal 
communication you exhibited 
contradicted your verbal 
message. 

Your use of supporting 
pnysical behaviors needs a lot 
ot work 



Poatura 

Watch posture-loaning on 
■eclurn 

Avoid one-logged posture 
Your pooi posture detracted a 

nrnal — /teal frnm uru i r fnnarh 



3 



Phyalcal Behevlore 

Good interaction with 
audience 

Place hands on tectum 

Leave pencil/pen at your desk 

Your non-verbal 
communication strengthened 
your verbal message to some 
extent. 

Try to keep papers quiet 

Keop papers flat 

Work for a more dynamic 
delivery. 

Your use of supporting physical 
behaviors needs a tittle work 

Your physical behaviors 
supported your verba! message 
adequately 

Poatura 

Dont btart speaking before 
assuming a speaker's position. 

Your postura was fine; it did 
not distract from your 



Phyalcal Bahavlora 

Ycur exceptional ohysical 
behaviors enhanced your 
credibility. 

Excellent use of nonverbal 
cues 

Your noteworthy non-verbal 
behaviors consistently 
supported your message very 
well. 



Poatura 

Your exceptional posture 
consistently uupporled your 
verbal message 

Your noteworthy posture 
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Appendix D 
Copy of Sample Printout from the 
Computerized Competent Speaker Evaluation System 
THE COMPETENT SPEAKER SPEECH PERFORMANCE EVALUATION FORM 



SPEAKER'S NAME: Chris Student 
EVALUATOH'S NAME: StevaWmac 

EIGHT PUBLIC SPEAKING COMPETENCIES 



ASSIGNMENT: Samp* Spwch 
DATE: Tot. Jul 5. taw 

SPEAKING PERFORMANCE RATING 

UNSAT SAT EXCELL 

0 0 0 



COMPETENCY ONE: Cl>ooaaa and narrow* a topic appropriate to tha 

audianoa and occasion 
COMMENTS: 

Excaflant topic auction E 

TkntJy topic satectJon E 

Chris, your topic about <ha uaa oi tha computer In tha classroom is fantastic for thia profac! Whara dW you coma 
up wtth tha Idaa? E 



COMPETENCY TWO: Communicates tha tha«is/»p#cif>c purposa In a 

mannof appropriate tor tha audianoa and occaaton 

COMMENTS: 

Your audianoa Hkad your apacHic purposa S 
Interasting ta-to of thaaia/spaciflc purpoaa with tha occasion S 



6 



COMPETENCY THREE: Providaa appropriate aupporting material baaad 0 0 7 

on lha audianoa and occasion 

COMMENTS: 

Outstanding visual akJt E 
Gooduaa of cradfcla aouroaa S 

Tha aupporting matarial you choaa fit tha occaaion wait. S 

Tha vlauaJ aida of your computer acraana raally anhanoad your apaach. but thay naadad to ba iargar for avaryona 
to aaa. S 



COMPETENCY FOUR: Uaaa an organizational pattern appropriate 
to topic, audianoa, occasion, and purpoaa 

COMMENTS: 

I undarstand. Your point! art waN provso. E 
Forcaful conctoaton E 

Tha body of your apaach ctearty comrourJcatea your maaaaga to audtenco E 
Tha organizational davaiopmant waa axtramaty appropriate for tha occaaion E 
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(Appendix D continued) 



COMPETENCY FIVE: Um language t*i it appropriate 0 0 9 

to tm audience tnd occation 

COMMENTS: 

Your terminology wee w»H- thought out and very appropriate for thit eudtence E 

Your um of Jargon w«e exceptionally taateful for thie occealon. E 

Your language waa exceptionally clear tnd tenaitive to nuancee of meaning. E 

You had a very 0<x>d balance between ttrmt ttal were diedptine tptctTic to your tpeech and termt that your 
audtence could understand. Chria. E 

COMPETENCY SIX: Uta« vocal variaty In rata, pitch, and Intensity 2 0 0 

to heighten and maintain Interact 

COMMENTS: 

Your voloa doaa not tound convarttlona]. U 

Try to alow down your rata of delivery. U 

Your pitch tound* too high-work on a fowar prtch. U 

Speak foudar and try to vary your intenerty to emphasize polntt. U 

Chria, can you coma to taa ma toma ttma toon to frat wa can talk about soma exercke* that you can do to work on 
She pitch of your voice? I would Ifce to tuggaat toma braathing exercises. You arc not alone in thit. t had a 
atudant a tew aamaatart ago who realty worked on pitch and mada graat improvement* I U 

COMPETENCY SEVEN: Utaa pronunctetto ., grammar, and articulatkxi 0 5 0 

appropriate to the designated audianca 

COMMENTS: 

Your grammatical arrort mada understanding your tpaach tlfghtly difficult S 
Your pronunciation for al of tha worda you utad toundad correct E 
PronourVantecadant ditagraamant it a flagrant •nor. U 

Your mltuaa of grammar tlightly da tr acted from your tpaach and tha occasion. S 



COMPETENCY EIGHT: U*ee phy»icai behavior* that •upport 
the varbal mactaga 

COMMENTS: 

Your phyafcal behaviors do not tupport your varbal maataga. U 
Donl atari tpaaklng bafora aaauming a apeaker'e position. S 
Your delivery it atjn and Jerky. U 
Try moving away from tectum S 

Your unacceptable facial expressions ara Incongruant with your varbal intent U 
I realty appreciate your affort to draaa wal for your tpaach. E 



GENERAL COMMENTS: TOTAL SCORE OF COMPETENCIES 4* 

Chria, I appreciate tha increased effort that you are putting into each tpeech. 
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Appendix E 

Sample Costs of Hardware Components of the 
Computerized Competent Speaker Evaluation System 
A very basic package for any communication 
department considering the use of the Computerized 
Competent Speaker Evaluation System is listed below. 
Since computer hardware goes through so many 
developments so quickly, the hardware that was 
purchased for this experiment in 1991 has already been 
outdated. Therefore, a computer analyst at the 
University of Colorado at Colorado was consulted 
(Witter, 1994a) . He indicated that the hardware used 
in this experiment was the middle-of- the- line 
Macintosh tm hardware at the time that it was purched. 
He suggested contemporary replacement hardware. An 
electrical engineer at. the Univerisity of Colorado at 
Colorado Springs was consulted about the cost of: 
installation (Witter, 1994b). Approximate 1994 are 
provi dcd : 

One Macintosh Quadra 6S0 tm 8MB Hard Disk 2 30 CPU 
(model M2107LL/A) - 

;^pp. $1,700.00 x 2 CPUs ; $1,400.00 

(Appendix E continues) 

63 



Computerized Speech Feedback 

63 



(Appendix E continued) 

One Apple tm Extended Keyboard 
(model M0312) « 

app. $140.00 x 2 keyboards = 

One Macintosh tm Color Display 
(model M1198LL/B) = 

app . $360.00 x 2 monitors = 

One Macintosh tm LaserWriter Pro 630 (8MB) 
(model M5858LL./B) = 



$280 .00 



$720 .00 



app. $1,645.00 x 1 printer = $1,645.00 

Installation of the two Macintosh tm 

computers with monitors and keyboards 
and the hardware for the network 
connections = 



app. $60.00 



1 inst . chg . 



$60 . 00 



Training of one person at $8.00/hour 
for 2 work days (16 hours) = 

app. $128. 00 x 2 raters 

Total 



$256 .00 
$6, 361 .00 



An optional and less expensive hardware package 
would consist of the following. The costs for the CPU, 
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(Appendix E continued) 

keyboard, monitor, installation, and training would be 

the same. A less expensive, but adequate printer 

would complete the package. 

CPU, keyboard, monitor, installation, 

and training = app . $4,716.00 

DeskWriter tm 520 

(Part ACZ1718) = app. $365.00 

Total $5,081.00 

Note: Costs have not been established for the purchase 
of the Computerized Competent Speaker Evaluation 
System . 
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Author ' s Note 
The primary researcher of this project would like 
to thank other individuals who contributed to the 
development of the Computerized Competent Speaker 
Evaluation System. In addition to Dr. Sherwyn P. 
Morreale, Dr. Donald D. Morley and Dr. Kim B. Walker 
served on the thesis committee for this project. Dr. 
Morley provided methodological direction, and Dr. 
Walker offered insightful criticism of this project. 

Graduate students were vitally involved in this 
project. Beth Zautke conducted separate interviews of 
Dr. Ralph Behnke and Dr. Chris Sawyer, traveling from 
Colorado to Texas to conduct those interviews. For 
several hours, Kathy Ellis, Josie Mares-Dean, and Lori 
Pinello-Tegtmeier sorted the comments in both of the Q 
sorts. Colleen McCormick and Marcel Hunter offered 
helpful suggestions as they evaluated students' 
speeches in the test group with the Computerized 
Competent Speaker Evaluation System. Laura Austin and 
Hazel Lowe rated students' speeches in the control 
group . 
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End Note 

x The terminology that refers to the microcomputer 
as a "tool" of mankind often alludes to the prevalent 
assumption in academia today that mankind has evolved 
from the primates. The primary researcher in this 
study categorically disagrees with this assumption. 
Reason, itself, points to a blatant discrepancy evident 
in the comparison of the development of the 
microcomputer systems of the later part of the 
twentieth century to the evolution of the world as it 
is known today. This so-called "tool" of humankind, a 
technological wonder of this century, has been 
developed by intelligent and purposeful engineers. 
Furthermore, the Computerized Competent Speaker 
Evaluation System was developed by a purposeful 
researcher . This experimenter disagrees with the 
notion that the computer is a "tool" that has been 
developed by humankind as he/she has evolved from the 
primates into humankind. 



Computerized Speech Feedback 

Table 1 

t tests ot Differences in Mean Scores on Speeches for 
Students Receiving Computerized Feedback as Compared to 
Students Receiving Handwritten Feedback 







St d. 


t 


2-tail 


Speech N 


Mean 


Pev, 


value 


d.f, Prob. 


Final 






3.0 


53.6* .004 


Comp Fdbk 2 9 


99. 3 


10 . 4 






Hdwtn Fdbk 28 


91 . 7 


8 . 5 







* B <.01 
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Table 2 

t tests of Differences in Mean Scores of the Raters on 
the Final Speech for Students Receiving Computerized 
Feedback as Compared to Students Receiving Handwritten 
Feedback 



Racers 




N 


Mean 




Std. 
Dev. 


t 

value 


d. 


f . 


2- tail 
Prob. 


Raters 1 


St 


2 








1 , 8 


17 


. 5 


. 095 


Rater 1 




13 


102 . 


6 


3.4 










Rater 2 




16 


96. 


6 


n .2 










Raters 1 


St 


3 








3 . 8 


20 


5 * * 


. 001 


Rater 1 




13 


102 . 


6 


3.4 










Rater 3 




14 


95. 


4 


6.3 










Raters 1 


St 


4 








5.6 


16 


9 * * * 


. 000 


Rater 1 




13 


102. 


6 












Rater 4 




14 


88 


1 


9.1 
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Tabl e 2 , cont inued 



Raters 2 & 3 .33 22.05 .747 

Rater 2 16 96 .6 13 .2 

Rater 3 14 95.4 6.3 



Std. t 2-tail 

Raters U Mean Dev. value d. f . Prob . 

Raters 2 & 4 2.1 26.6* .047 

Rater 2 16 96.6 13.2 

Rater 4 14 88.1 9.1 



Raters 3 & 4 2.5 23.1* .021 

Rater 3 14 95.4 6.3 

Rater 4 14 88.1 9.1 



Raters 1,2 & 3,4 3.0 53.6** .004 

Raters 1,2 29 99.3 10.4 

Raters 3,4 28 91.7 8.5 



* g<.05 
** k<-01 
*** g< . 001 
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Table 3 

i Loom ui DlUmmicuu in llemu vn m ukl « j IL 
Questionnaire for Students Receiving Compu terized 
Feedback and Students Receiving Handwrit ten Feedback 



Questionnaire N 
Item 


Mean 


Std. 
Dev. 


t 

value 


d . f . 


2-tail 
Prob . 


Lapsed Time (1) 






4 . 3 


4(> . SO*** 


.000 


Comp Fdbk 27 


5.8 


1 .3 








Hdwtn Fdbk 2 7 


4 . 0 


1 . 8 








Evaluation by 

Commit er Bothersome (5) 




2 . 6 


51.0* 


.012 


Comp Fdbk 27 


2 .5 


1 . 7 






Hdwtn Fdbk 27 


3 . 9 


1 .4 








Database of Comments (8) 




-2.1 


47.6* 


. 044 


Comp Fdbk 27 


3 .7. 


1 .7 








Hdwtn Fdbk 27 


2 . 9 


1 . 2 
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Time Frame in which 

Feedback Given was Good (13) -2.7 45.2* .011 

Comp Fdbk 27 5.7 1.0 

Hdwtn Fdbk 27 4.7 1.6 



Std. t 2-tail 

Questionnair e N Mean Dev . value d. f . Prob . 

Item 

Stan . /Non - Stan . Comments 

Allow Qbi./Subi. Grading (14) -2.7 51,9** .009 

Comp Fdbk 2 7 4.7 1.3 

Mdwt n Fdbk 27 3.7 1.3 



* p<.0S 
** b<.01 
*** £< . 001 
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Table 4 

Observed Mean and Adjusted Mean of Underlying 
Constructs in Questionnaire with Grouping of Students 
by Confounding Variable 

Source of Variation Observed Adjusted 

Mean Mean 

Personalized Comments (Database) 
(Items 3,91 

Comp Fdbk 3.60 3.60 

Hdwtn Fdbk 2.96 2.96 

Liking for Computerized 
/Handwritten Feedback 
(Items 4,10) 

Comp Fdbk 4.88 4.89 

Hdwtn Fdbk 5.31 5.31 

Type of Feedback Limited Rater 
(Items 6,10) 

Comp Fdbk 3 . 50 3.51 

Hdwtn Fdbk 3.69 3 .68 

Rater/Feedb.ack/Time Frame- Improvement 
(Items 1,9,12,13) 

Comp Fdbk 4.61 4 .63 

Hdwtn Fdbk 3.57 3 . 5S 
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Table b 

Multivariate Analysis of Covariance o f Underlying 
Constructs in Questionnaire by P erceived Grade in COMM 
210 



Source o£ Variation d . f . MS 

I't* ruona 1 i *z«'«i Comments (Databdtio ) 
(Items 3,8) 

1 5.34 

hiking fen Computerized 
/Handwritten Feedback 
(Items 4,10) 

1 2.31 



Type of Feedback Limited Rater 
(Items 6 , 10) 

1 0.38 0.19 .668 

Rater/Feedback/Time Frame = 

Improvement 

( Items 1,9,12,13) 

1 14.99 5.41* .024 



* * £< . 01 
*** £<.00l 



F Level of 

Sis 



3.10 .084 



1 . 30 .260 
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